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(54) Detecting material failures in ground locations 



(57) A method for capturing images of ground loca- 
tions and for detecting the presence of failure(s) or ma- 
terial failures in man-made structures in such ground lo- 
cations is disclosed. The method provides an" image 
sensor spaced remotely from the -ground and' which se- 
quentially captures a number of images of various 
ground locations to provide digital images; processing 



captured digital images to determine the presence of a 
potential material failure in a man-made structure in ac- 
cordance with predetermined coordinate positions 
. which locate the man-made structures .in one or more 
of the captured digital images; and indicating to a cus- 
tomer that a potential material failure has been detected 
in a predetermined coordinate position. 
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Description 

[0001] The present invention relates to remotely de- 
tecting material failures in a ground location by using an 
image sensor. 

[0002] The surveillance of ground topography is well 
known in the art. It is frequently the case that an aircraft 
or a satellite includes an image capture device such as 
a CCD. In ground surveillance it is highly desirable to 
detect whether there has been a material failure in a 
man-made object such as a road, a pipeline, an electri- 
cal grid, or other man-made structures of practical inter- 
est. When detected, a determination is made if remedial, 
action must betaken. Often times a visual inspection of 
ground topography is provided by a land-based crew 
that traverses an area by vehicle or foot, to determine if 
there is a material failure. Airborne photographic sys- 
tems can also be used for capturing images of adjacent 
areas. These images are then reviewed to determine if 
there is a material failure. 

[0003] It is an object of the present invention to pro- 
vide an improved way to automatically determine if there 
is a material failure in a man-made structure. 
[00Q4] This object is achieved by a method for cap- 
turing images of ground locations and for detecting the 
presence of material failures in man-made structures in 
such ground locations comprising the steps of: 

(a) providing an image sensor spaced remotely 
from the ground and which sequentially captures a 
number of images of various ground locations to 
provide digital images; 

(b) processing captured digital images to determine 
the presence of a potential material failure in a man- 
made structure in accordance with predetermined 
coordinate positions which locate, the man-made 
structures in one or more of the captured digital im- 

5 ages; and ■ j ■ 

(c) indicating to a customer that a potential material 
failure has been detected in a predetermined coor- 
dinate position. 

[0005] In many cases it is required to inspect man- 
made structures frequently in order to determine the 
likelihood or development of potential material failures. 
In many cases these inspections are done by a ground 
site survey; individuals visit these locations and take 
measurements or other form of data on-sight. This proc- 
ess becomes cumbersome, costly, inconvenient, and in 
many cases unreliable and unsafe due to the dangers 
present in remote locations and to potential false inter- 
pretations due to worker fatigue and other factors. Fur- 
thermore, remote locations are frequently in mountains, 
deserts and forests that are difficult to reach and fre- 
quent inspection requires the placement of permanent 
maintenance and inspection crews adding to overall 
costs. It is an advantage of the present invention to pro- 
vide a more effective way of determining material fail- 



ures in man-made structures by automatically process- 
ing images captured from a remote platform. This auto- 
matic processing can include comparing with previously 
detected images. This automatic processing can also 
5 include algorithms and expert systems that act in a pre- 
dictive manner. 

[0006]' A feature of the present invention is that a 
chemical agent or a chemical change associated with a 
material failure can be used in a man-made structure of 
10 practical interestthat is particularly suitable for detection 
after a material failure. 

FIG. 1 depicts a system for capturing images from 
an airborne or a satellite platform in accordance 
15 withthe present invention; 

FIG. 2 is a flowchart in block diagram form of the 
process of capturing and processing images to de- 
tect potential material failures in man-made struc- 
tures; 

20 'FIG.- 3ris a. flowchart in block diagram form of an 
image processing algorithm which can be used in 
the system shown in FIG. 1 ; 

FIG .. 4 illustrates a somewhat different embodiment 
of the present invention that uses a chemical agent 
25 to aid in detecting potential material failures in a 
man-made object; and 

FIG . 5 illustrates the capturing of an image, analysis 
; to identify a potential material failure, and commu- 
nication over a channel to deliver information to and 
30 receive; payment from a customer., , 

[0007] A-sensor system- 1 employed in the capturing 
of images- in order to identify materia! failures in man- 
made structures may be mounted on either an aerial or 

35 a satellite platform. Images of the ground containing var- 
ious man-made structures such as roadways, pipelines, 
electrical power lines, agricultural, mining, real estate 
activity and the like are captured by this sensor system! . 
The term "man-made structure" can also include other 

40 human activities such as insecticide spraying which af- 
ter application can be detected by sensor system 1 . In 
such a case, an additive can be included in the insecti- 
cide spray that can be detected by the image sensor. 
Sequential: images may be captured in digital form and 

45 either stored in the aerial or satellite platform to be trans- 
ferred later or transmitted via a radio link to a control 
ground station. The capture device 2 includes an elec- 
tronic sensor, typically a CCD or CMOS imaging array 
that along with some imaging optics captures a picture 

50 of the scene in electronic form. In some instances a spe- 
cial optical filter 3 is attached to the input to the CCD or 
CMOS detector to filter the light wavelengths which are 
incident upon the detector.. This optical filter 3 is chosen 
so as to maximize the signal-to-noise ratio for the de- 

55 tection of a specific type of materia! failure. Alternatively, 
the ground location image can be captured by conven- 
tional photographic cameras. Film images would then 
have to be converted to digital images by an image 
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scanner that includes an image sensor. The system 1 
also has an image capture control circuit4 that sequenc- 
es the operation of the capture device 2: As will be clear 
from FIG. 1 , the operation of the various elements 
shown in system 1 are under the control of a control 5 
computer 31 . The image capture control circuit 4 con- 
trols capture device 2 and sends position and orientation 
information to a position and orientation storage circuit 
5 with each captured image. Position information in the 
form of spatial coordinates is provided by the customer 10 
in order to identify the location of man-made structures 
of interest. Such position information is also stored in 
position and orientation storage circuit 5v Position and 
orientation data are used along with predetermined co- 
ordinate positions to locate the man-made structures in is 
the captured image. Control computer 31 causes image 
data to be stored in image storage 6 and can be proc- 
essed to identify features of a scene in image process- 
ing circuit 7. The processing sequence is also directed 
by control computer 31 of the image data in this instance 20 
is to enhance the capability ofthe system 1 to identify 
material failures in man-made structures. The image 
processing circuit 7 includes a storage memory (not 
shown) that includes a representation- of different mate- 
rial failures to be detected and comparing the captured 25 
digital image with the material failures to.determine the 
presence of a material failure, type of material failures 
and location of the material failures. 1 With the exception 
ofthe capture device 2, the various .elements 'of the sys- 
tem 1 may be located either in the remoteplatform or at 30 
the ground station location. Moreover, many of the ele- 
ments described can be embodied in- software Which 
can be understood to be within the control computen31 . 
The capture device 2 is located in either the aerial or 
satellite platform "or a fixed structure spaced above the 35 
ground. -.'-*-.■' v. * \ 

[0008] The overall process for detecting material fail- 
ures in man-made structures is depicted in flowchart 
form in FIG. 2. The flowchart is in block diagram form 
and those skilled in the art will appreciate that many of *o 
the functions are controlled by the control computer 31 . 
The starting event includes initializing the capture de- 
vice 2 and image storage 6 to erase any previously cap- 
tured scene data. Next a new scene is captured in block 
9 using the position information supplied by the custom- 45 
er to trigger recording of the images. The image data 
along with position and time information necessary to 
identify the location and time of the. current scene is 
stored in order to facilitate comparison with the same 
scene taken at other times. Image* and other data- are so 
stored in a scene' database 10 inorderto perform such 
comparisons at a futures time..: Image analysis 11 is next 
performed in order to identify changes in the scene and 
facilitate identification of material failures in the man- 
made structures that appear in the scene. The latest 55 
scene image is compared with image data that has been 
previously stored in the scene database 1 0. If a material 
failure is not detected the process stops. Detection of a 



material failure may initiate further image analysis 12 as 
required by a customer 13. The identification process 
finishes with the results of the analysis communicated 
to the customer 13. The communication make take 
many forms, for example a telephone contact or e-mail 
notification of the detection of the material failure. The 
final step in the process is to correct the material failure. 
[0009] FIG. 3 depicts the algorithm used to process 
image data files from a database and identifies material 
failures if they have occurred. Two separate, data files, 
scene (1)14 and scene (2) 15., are made available for 
comparison. Both data files contain the same scene 
content, but they typically record images taken at differ- 
ent times. That is, the time between capturing the two 
images differs by a time At. Both image files or scenes 
undergo the process of ort ho rectification 16, that is, 
compensation for variations in position and angle at the 
time the scenes were recorded. This process is per- 
formed in order to allow an exact pixel by pixel compar- 
ison of the elements of a scene or image. It may or may 
not be necessary to correct the data in each scene for. 
differences in the illumination 1 7 at the time each scene 
was recorded. Changes in the scene are identified in 
block 1 8 are used by the control computer 31 by detect- 
ing, using software, differences in ; the pixel content, of 
the two scenes to be compared. Such changes may be 
reflected in the intensity of the pixels, or in the shape of 
an object, corresponding to a finite collection of pixels. 
Such methods for identification of pixel or object chang- 
es.are well known to those skilled in the art. On the basis 
of such pixel changes the material failure type is identi- 
fied in block 1 9 and the customer 1 3 is notified 20 of the 
existence of the failure. 

[0010] FIG. 4 illustrates a somewhat different embod- 
iment of the present invention that uses an agent to aid 
in detecting material failures in a man-made object. An 
aerial platform 21 performs image capture 22 of a man- 
made structure (in this case a roadway, 23) in the man- 
ner previously described, FIG. 4 also shows image cap- 
ture of a.roadway 23 with an isolated man-made failure 
24. In this instance, the detection of the man-made ma- 
terial failure is enhanced by the presence of a chemical 
or physical image contrast-enhancing agent. The cop? 
trast agent for example, could be released to the envi- 
ronment as a consequence ofthe material failure occur- 
ring. For example, encapsulated fluorescent dyes, eir 
ther as isolated molecular species or in crystalline form, 
embedded in a roadway could be released with crack 
formation. Triboluminescent materials that emit light up- 
on mechanical impact could be sensed to determine the 
likelihood of material failure as a result of mechanical 
impact. Yellow-emitting rhalophosphate phosphors are 
well established as sensors when used in this mode of 
operation. In another use scenario, the contrast agent 
is incorporated into a man-made structure in a form 
where it becomes activated upon the occurrence of a 
material failure. For example, certain chemical species 
may undergo a color change upon exposure to air or 
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other chemical species. An example is the use of corro- 
sion-sensitive paints applied to pipelines. Color-chang- 
ing chemical compounds (indicators) such as phenol- 
phthalein or bromothymol blue, or fluorescing chemical 
compounds such as coumarins, can be incorporated in- 
to acrylic paints and applied to pipelines. Corrosion of 
the pipe causes a pH change in the paint; the conse- 
quence of which is a color change in the indicator or flu- 
orescence. Such color changes are recorded remotely. 
Color-change materials exist that are sensitive to 
changes in pH, oxygen concentration, and the presence 
of trace quantities of various metal ions in the environ- 
ment (chelating agents). The use of such materials is 
well known to those skilled in the art, and may be used 
singly or in combination in remote sensing applications. 
[001 1] Such materials in combination with the optical 
filter 3 in FIG. 1 significantly improve the signal-to-noise 
ratio, and hence the detectability for remote sensing. 
The combination of the optical filter transmission func- 
tion and the color-change to be detected as a result of 
a material failure will be optimized for each application. 
This optimization is well known to those skilled in the art. 
[0012] FIG 5 illustrates the capturing of an image, 
analysis to identify a material failure and communication 
over a computer network to deliver information to and 
receive payment from the customer 1 3. A satellite 25 or 
an aerial platform 26 captures an image of a scene 27 
that contains a man-made structure (in this case elec- 
trical utility lines) to be analyzed. The image data is 
transmitted to a ground station 28 and transferred to the 
service provider's computer system 29. The image data 
is analyzed as previously described to determine wheth- 
er a material failure has occurred. If a failure is detected 
the customer 13 for the service receives notification of 
the failure via a channel for example, a computer net- 
work such as the Internet, or via other means, such as 
telephony. The customer computer 30 receives the no- 
tification directly from over the computer network. The 
customer 1 3 subscribes to the service and pays for the 
service via the computer network. In this manner, the 
timely delivery of information regarding the status of a 
failure can be transmitted to the customer and the qual- 
ity of service can be assured to be at a sufficiently high 
[001 3] Other features of the invention are included be- 
low. 

[0014] The method further including providing a 
chemical agent that includes which materials which 
when released reacts with substances in the ground to 
provide a detectable material failure to the image sen- 
sor. 

[0015] The method wherein the image processing in- 
cludes comparing previously captured images with new- 
ly captured images to determine variations in a ground 
condition which could contain the material failure. 



Claims 

1 . A method for capturing images of ground locations 
and for detecting the presence of material failure(s) 

5 or failures in man-made structures in such ground 

locations comprisingthe steps of: 

(a) providing an image sensor spaced remotely 
from the ground and which sequentially cap- 
tures a number of images of various ground lo- 
cations to provide digital images; 

(b) processing captured digital images to deter- 
mine the presence of a potential material failure 
in a man-made structure in accordance with 
predetermined coordinate positions which lo- 
cate the man-made structures in one or more . 
of the captured digital images; and 

(c) indicating to a customer that a potential ma- 
terial failure has been detected in a predeter- 
mined coordinate position. 

2. The method of claim 1 further including: 

(d) sending captured processed digital images 
with detected potential material failures to a 
customer. 

3. The method of claim 1 wherein the digital image 
processing includes.comparing previously captured 
digital images with newly captured digital images to 
determine variations in the captured digital images 

'■ at the predetermined coordinates which indicate a 
potential material failure in a man-made. structure. 

4. The method according to claim 1 wherein the digital 
images are captured by a capture device which is 
located in a fixed structure position above the 
ground location or in a moving structure such as an 
aircraft or satellite. 

5. The method of claim 3 wherein the image process- 
ing includes storing in memory a representation of 

. different material failures to be detected and com- 
paring the captured digital image with the material 
failures to determine the presence of a material fail- 
ure, type of material failures and location of the ma- 
terial failures. 

6. - A method for capturing images of ground locations 
and for detecting the presence of material failure(s) 
or failures in man-made structures having a detect- 
able chemical agent in such ground locations com- 
prising the steps of: 

(a) providing an image sensor spaced remotely 
from the ground and which sequentially cap- 
tures a number of images of various ground lo- 
cations to provide digital images; 
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(b) processing captured digital images to deter- 
mine changes in the chemical agent which in- 
dicate the presence of a potential material fail- 
ure in a man-made structure in accordance with 
predetermined coordinate positions which lo- 
cate the man-made structures in one or more 
of the captured digital images; and 

(c) indicating to a customer that a potential ma- 
terial failure has been detected in a predeter- 
mined coordinate position. 

7. The method of claim 6 wherein the chemical agent 
includes materials which when leaked from a recep- 
tacle are adapted to be detected. 

8. The method of claim 6 wherein the chemical agent 
includes materials which when released react with 
substances in the ground to provide a detectable 
material failure to the image sensor. 

9. A method for capturing images of ground locations 
and for detecting the presence of material failure(s) 
or failures in man-made structures in such ground 
locations comprising the steps of: 

(a) providing an image sensor spaced remotely 
from the ground and which sequentially cap- 
tures a number of images of various-ground lo- 
cations to provide digital images; 

(b) processing captured digital in ages to deter- 
- mine the presence of a potential material failure 

in a man-made structure in accordance with 
predetermined coordinate positions which lo- 
cate the man-made structures in one or more 
of the captured digital images; 

(c) indicating to a customer that a potential ma- 
terial failure has been detected in a predeter- 
mined coordinate position; and 

(d) correcting material failures. 

10. A method for capturing images of ground locations 
and for detecting the presence of faiiure(s) or ma- 
terial failures in man-made structures in such 
ground locations and making payment for the de- 
tection or correction of detected material failures 
comprising the steps of: 

(a) providing an image sensor spaced remotely 
from the ground and which sequentially cap- 
tures a number of images of various ground lo- 
cations to provide digital images; 

(b) processing captured digital images to deter- 
mine the presence of a potential material failure 
in a man-made structure in accordance with 
predetermined coordinate positions which lo- 
cate the man-made structures in one or more 
of the captured digital images; 

(c) indicating to a customer that a potential ma- 
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terial failure has been detected in a predeter- 
mined coordinate position; 

(d) correcting material failures; and 

(e) making payment for the material failure. 
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